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FIELD OF THE INVENTION: 
The invention relates to image interpolation, particularly to using spatial 
interpolation to interpolate the re-sampled lines in an image, including motion pictures. 

BACKGROUND: 

As television moves from an analog system to a digital system, several problems 

arise. 

When the input image size and the display size are different, a resample 
processing which creates the pixel at a non-lattice position is needed. Also, in order to 
display interlaced video signals on a progressive display with high visual quality, it is 
necessary to 'de-interlace" interlaced odd and even fields of video signals into a 
sequentially continuous field of pixels without (inter-line) gaps. 

Specifically, the most common method of implementing the interpolation function 
is to use a 2-line memory and generate the interpolated pixels from the pixel above 
(previous line) and the pixel below (next line). This implementation can be improved by 
using more lines to implement a higher order interpolation filter, giving slightly smoother 
images. However, those filters always operate on the basis that the current pixel is related 
to the pixel above and below it, which is completely untrue of edges with small angle. 
Whenever an edge is not vertical, the current pixel is related to those above and below it 
along the direction of the edge, e.g. if the angle is 45 degree the relationship is one up and 
one across and one down and one across. With small angle becomes more complex and 
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will involve non-integer relationships. Moreover, currently prior art approaches only 
implement a simple angle interpolation; nothing is done on small angle edges. 
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BRIEF DESCRIPTION OF THE FIGURES: 
The accompanying drawings which are incorporated in and form a part of this 
specification, illustrate embodiments of the invention and together with the description, 
serve to explain the principles of the invention: 

Figure 1 A shows an in portion of an image to be resampled accordance with one 
embodiment of the invention. 

Figure IB shows possible interpolation directions established in the portion 
(shown in Figure 1 A) in accordance with the embodiment of Figure 1 A. 

Figure 1 C shows pixel domains associated with a possible interpolation direction 
in accordance with the embodiment of Figure 1 A. 

Figure 2 is a flow chart outlining steps performed for interpolation in accordance 
with one embodiment of the invention. 

Figure 3 depicts a system for performing spatial interpolation in accordance with 
one embodiment of the invention. 
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DETAILED DESCRIPTION: 
Reference is made in detail to the preferred embodiments of the invention. While 
the invention is described in conjunction with the preferred embodiments, the invention is 
not intended to be limited by these preferred embodiments. On the contrary, the 
invention is intended to cover alternatives, modifications and equivalents, which may be 
included within the spirit and scope of the invention as defined by the appended claims. 
Furthermore, in the following detailed description of the invention, numerous specific 
details are set forth in order to provide a thorough understanding of the invention. 
However, as is obvious to one ordinarily skilled in the art, the invention may be practiced 
without these specific details. In other instances, well-known methods, procedures, 
components, and circuits have not been described in detail so that aspects of the invention 
will not be obscured. 

Referring now to Figure 1 A, a portion of an image 100 that is to be resampled by 
spatial interpolation is shown in according to one embodiment of the invention. As 
shown, image 100 comprises lines 110 and 120 that are separated by a resampling line 
115. 

Resampling is performed on resampling line 1 15 of image 100. Specifically, line 
1 10 comprises pixels x[-5] to x[13]; line 120 comprises pixels y[-5] to y[13]. Pixel x[4] 
is located immediately above a resampling pixel 111 (in resampling line 115); pixel y[4] 
is located immediately below resampling pixel 111. 
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In order to resample image 100, spatial interpolation is performed at the position 
of resampling pixel 1 1 1 to provide a pixel value to position 111. This spatial 
interpolation is performed along a direction that is selected from a set of possible 
interpolation directions (to be described in Figure IB) intercepting at position 111. 

Referring now to Figure IB, interpolation directions D[0] to D[8] for performing 
spatial interpolation are shown in accordance with one embodiment of the invention. 
Spatial interpolation for position 1 1 1 can be performed along one of interpolation 
directions D[0] to D[8]. Directions D[0] to D[8] intercept effectively at the center of 
position 111, thereby forming various angles with respect to resampling line 115. 

Specifically, as shown in Figure IB, direction D[0] is defined by a line effectively 
intercepting the centers of pixels x[0] and y[8]. Direction D[l] is defined by a line 
effectively intercepting the centers of pixels x[l] and y[7]. Direction D[2] is defined by 
a line effectively intercepting the centers of pixels x[2] and y[6]. Direction D[3] is 
defined by a line effectively intercepting the centers of pixels x[3] and y[5]. Direction 
D[4] is defined by a line effectively intercepting the centers of pixels x[4] and y[4]. 
Direction D[5] is defined by a line effectively intercepting the centers of pixels x[5] and 
y[3]. Direction D[6] is defined by a line effectively intercepting the centers of pixels x[6] 
and y[2]. Direction D[7] is defined by a line effectively intercepting the centers of pixels 
x[7] and y[l]. Direction D[8] is defined by a line effectively intercepting the centers of 
pixels x[8] and y[0]. 
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Moreover, interpolation directions D[0] to D[8] can be classified into three 
different groups. Directions D[0] to D[3] belong to the group of directions having 
negative slopes. Direction D[4] belongs to the group of infinite slope (i.e., the vertical 
direction). Directions D[5] to D[8] belong to the group of directions having positive 
slopes. 

Edge detection is performed along directions D[0] to D[8]. Once a possible edge 
is detected along a particular direction out of directions D[0] to D[8], such direction is 
selected to be the interpolation direction. Then, verifications are performed to make 
certain that indeed an edge is lying along the selected interpolation direction. In turn, 
once the selected interpolation direction is verified to contain an edge, spatial 
interpolation is performed using pixels lying along the selected interpolation direction. 
On the other hand, if the selected interpolation direction does not pass the verifications, it 
is disqualified as the interpolation direction. Instead, a default interpolation direction is 
used for performing spatial interpolation. 

Referring still to Figure IB, numerical values A[0] to A[8] are assigned 
respectively to interpolation directions D[0] to D[8]. Then, based on A[0] to A[8], a 
selection method then selects a possible interpolation direction from directions D[0] to 
D[8]. Specifically, A[0] to A[8] quantify the likelihood of having an edge lying 
respectively along directions D[0] to D[8], wherein the more likely an edge lies along a 
direction, the smaller the numerical value assigned to that direction. As such, an 
interpolation direction is selected from directions D[0] to D[8] as the direction having the 
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minimum numerical value. 

As understood herein, the possible interpolation directions need not be limited to 
the nine directions D[0] to D[8] as shown here. For example, in another embodiment, 
additional directions can be defined intercepting position 111, wherein even smaller 
angles are formed between the additional directions and resampling line 115. On the 
other hand, in yet another embodiment of the invention, less number of directions can be 
defined intercepting position 111, wherein bigger angles are formed between the 
intercepting directions and resampling line 115. 

In the present embodiment, for k = 0 to 8, numerical value A[k] is defined as 



i = k-4 to k+4, and j = 8-k-4 to 8-k+4, wherein P is a function that assigns a pixel value to 
a pixel. Specifically, for k = 0 to 8, A[k] quantifies the degree of similarity between two 
segments of pixels, wherein these two segments are symmetric about position 111. This 
pair of symmetric segments are seg[k,x]={x[k-4], x[k-3], .. . , x[k+4]} and seg[k,y]={y[8- 



A[k] = S|P(x[i])-P(y[j]H 



k-4],...,y[8-k+4]}. 



Using A[k=l] as an example, 



A[l] = S|P(x[i])-P(y[j]) 
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for i = -3 to 5, and for j = 3 to 1 1. A[l] quantifies the similarity between two segments of 
pixels, wherein these two segments are symmetric about position 111. This pair of 
symmetric segments are seg[l,x]={x[-3], x[-2], ... , x[-5]} and seg[l,y]={y[3], y[4], 



As understood herein, a pixel value assigned by function P can refers to a gray- 
level value, a luminance value, a chrominance value, a composite signal value, or a 
combination of any channel of any image representation. 

Referring now to Figure 1C, two pixel domains (dom[l,x] and dom[l,y]) 
associated with interpolation direction D[l] is shown in accordance to one embodiment 
of the invention. Pixel domains dom[l,x] and dom[l,y] are used for respectively plotting 
graph[l,x] and graph[l,y] assigned to D[l]. Specifically, dom[l,x] includes pixels x[ -2] 
to x[ 4] from line 110; dom[l,y] includes pixels y[ 4] to y[10 ] from line 120. 

Although not shown in Figure 1C, each of D[0], and D[2] to D[8] also has its 
associated pair of pixel domains. In turn, each of D[0] to D[8] has its associated pair of 
graphs that plot pixel values vs. pixel position. The number of pixels in these pixel 
domains can vary according to which of D[0] to D[8] is selected. Specifically, for k = 0 
to 8, D[k] is associated with pixel domains dom[k,x] and dom[k,y]. More specifically, 
for k = 0 to 4, pixel domain dom[k,x] includes pixels x[ k-(4-k)] to x[ 4], and pixel 
domain dom[k,y] includes pixels y[ 4] to y [ (8-k)+(4-k)]. For k = 4 to 8, pixel domain 
dom[k,x] includes pixels x[4] to x[ (8-k)+(4-k)], and pixel domain dom[k,y] includes 



y[ii]}. 
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pixels y[ k-(4-k)] to y [4]. In turn, graph[k,x] and graph[k,y] show pixel values plotted 
respectively against dom[k,x] and dom[k,y]. 

As an example, D[0] is associated with pixel domains dom[0,x] and dom [0,y]. 
Pixel domain dom[0,x] includes pixels x[ -4] to x[ 4] and pixel domain dom[0,y] includes 
pixels y[ 4] to y [ 12]. In turn, graph[0,x] and graph[0,y] show pixel values plotted 
respectively against dom[0,x] and dom[0,y]. 

As another example, as the associated pixel domains for D[4], pixel domain 
dom[4,x] includes pixel x[4] and pixel domain dom[4,y] includes pixel y[4]. In turn, 
graph[4,x] and graph[4,y] show pixel values plotted respectively against dom[4,x] and 
dom[4,y]. 

As yet another example, as the associated pixel domain for D[6], pixel domain 
dom[6,x] includes pixels x[4] to x[8] and pixel domain dom[6,y] includes pixels y[0] to 
y[4 ]. In turn, graph[6,x] and graph[6,y] show pixel values plotted respectively against 
dom[6,x] and dom[6,y]. 

As will be described later with reference to Figures 2-3, once an interpolation 
direction (e.g., D[m]) has been selected, a peak/valley count test is performed using 
graph[m,x] and graph[m,y] associated with D[m], Peak/valley test exemplifies one of 
possible tests in performing segment analysis on one or both of symmetric segment pair 
associated with D[m]. 
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Specifically, the peak/valley count test is performed to count the number of peaks 
and valleys in graph[m,x], as well as the number of peaks and valleys in graph[m,y]. 
More than one peak or valley in either one of the two graphs indicates that an edge is not 
likely to be lying along direction D[m]. As such, according to the peak/valley count test, 
if more than one peak or valley is found in either graph[m,x] or graph[m,y], then D[m] is 
disqualified as the correct interpolation direction. Once D[m] is disqualified as such, 
spatial interpolation will not be performed along D[m], but rather along a default 
interpolation direction. 

Referring now to Figure 2 in view of Figures 1A-C, a flow chart 200 is shown 
outlining steps for performing spatial interpolation in accordance with one embodiment 
of the invention. The present embodiment allows spatial interpolation to be performed at 
various small angles that are not available in prior art approaches. Specifically, spatial 
interpolation for position 1 1 1 can be performed along one of interpolation directions D[0] 
to D[8]. Directions D[0] to D[8] form various angles with respect to resampling line 155. 

Edge detection is performed along directions D[0] to D[8] using symmetric 
segment pair detection. Specifically, once a possible edge is detected along a particular 
direction out of directions D[0] to D[8], such direction is selected to be the interpolation 
direction. Then, using symmetric segment pairs analysis and segment analysis, 
verifications are performed to make certain that indeed an edge is lying along the selected 
interpolation direction. In turn, once the selected interpolation direction is verified to 
contain an edge, spatial interpolation is performed using pixels lying along the selected 
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interpolation direction. On the other hand, if the selected interpolation direction does not 
pass the verifications, it is disqualified as the interpolation direction. Instead, a default 
interpolation direction is used for performing spatial interpolation. Finally, post 
processing can be performed on interpolated result to refine image quality. 

Steps 205, 210, 215 and 220 are performed for symmetric segment pair detection. 

In step 205, directions D[0] to D[8] are established using pixels x[0] to x[8] of 
line 1 10, and pixels y[0] to y[8] of line 120. D[0] to D[8] are possible directions along 
which to detect an edge intercepting position 111. 

In step 210, numerical values A[0] to A[8] are assigned to D[0] to D[8]. A[0] to 
A[8] quantify the respective likelihoods of detecting an edge lying along D[0] to D[8]. 
The smaller a numerical value is assigned to a possible interpolation direction, the more 
likely an edge is lying along such interpolation direction. 

In step 215, A[m], the minimum element of {A[0], ... , A[8]}, is detected. 

In step 220, D[m], the interpolation associated with A[m], is selected to be the 
direction most likely to detect an edge. 

Steps 225 and 230 are performed for symmetric segment pairs analysis. 

In step 225, a cross check test is performed. Performing such cross test can rule 
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out the problematic scenario of interpolating at position 111 along D[m] while two edges 
cross each other at position 111. Specifically, the test is performed to rule out a scenario 
wherein A[h] of direction D[h] (whose slope has the opposite sign from the sign of the 
slope of D[m]) is slightly difference from A[m]. The cross check test is also performed 
to rale out a scenario wherein the vertical direction A[4] is slightly difference from A[m]. 
D[m] passes the cross check test if no problematic scenario is found. 

As understood herein, for symmetric segment pairs analysis, an extreme value 
analysis can be performed in step 225 to replace the cross check test. For symmetric 
segment pairs analysis, the extreme value analysis can also be performed in step 225 in 
addition to the cross check test. The extreme value analysis is performed on the graph of 
A[t] for t varying continuously from 0 to 8. Specifically, the extreme value analysis is 
performed to detect any one of three cases. The first case is that A[m] is a minimum 
where m is not equal to 0 or 8, and the slope of graph of A[t] changes sign only once 
from negative to positive. The second case is that A[m] is a minimum where m is 0, and 
the sign of slope in graph of A[t] remains positive. The third case is that A[m] is a 
minimum where m is 8, and the sign of slope in the graph of A[t] remains negative. D[m] 
is disqualified if A[m] does not fit any one of the above three cases. 

In query step 230, a check is performed to verify if D[m] passes the cross check 
test. If affirmative, then step 235 is performed. Otherwise, step 289 is performed to 
interpolate along a default direction. 
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Steps 235, 240, 245 and 250 are performed for segment analysis. As understood 
herein, segment analysis does not need to include all of these four steps. For example, in 
one embodiment of the invention, segment analysis comprises steps 235 and 240, but 
without steps 245 and 250. In yet another embodiment, segment analysis comprises 245 
and 250, but without steps 235 and 240. 

In step 235, a peak/valley count test is performed on two graphs of function P 
obtained by applying P over two domains of pixels, one domain from line 110 and one 
domain from line 120. The peak/valley count test is used to analyze the two pixel 
segments from symmetric segment pair associated with D[m]. 

A peak is defined as a point on a graph having a slope on the left side of the point, 
and a slope on the right side of the peak point. Specifically, the left slope and the right 
slope meet at the point. The left slope is positive and larger than a pre-defined threshold; 
the right slope is negative and smaller than a pre-defined threshold. 

A valley is defined as a point on a graph having a slope on the left side of the 
point, and a slope on the right side of the valley point. Specifically, the left slope and the 
right slope meet at the point. The left slope is negative and smaller than a pre-defined 
threshold; the right slope is positive and larger than a pre-defined threshold. 

Specifically, D[m] is assumed to be D[l] as an example. For D[m=l], function P 
is applied to pixels x[-2] to x[4] of dom[l,x] to obtain pixel values P(x[-2]) to P(x[4]), 
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thereby resulting in a first graph, graph[l,x]. Additionally, function P is applied to pixels 
y[4] to y[10] of dom[l,y] to obtain pixel values P(y[4]) to P(y[10]), thereby resulting in a 
second graph, graph[l,y]. The peak/valley count test is performed to count the number of 
peaks and valleys in the first graph, as well as the number of peaks and valleys in the 
second graph. 

D[l] passes the peak/valley count test if no more than one peak or valley is 
presented in graph[l,x] and if no more than one peak or valley is presented in graph[l,y]. 
Specifically, more than one peak or valley in either one of the two graphs indicates that 
an edge is not likely to be lying along direction D[m=l]. As such, if more than one peak 
or valley are found in graph[l,x], or if more than one peak or valley are found in 
graph[l,y], then D[m=l] fails the peak/valley count test. Spatial interpolation will not be 
performed along D[m=l] if D[m=l] does not pass the peak/valley count test. 

To explain with another specific example, D[m] is assumed to be D[0]. 
Specifically, for the example of D[m=0], function P is applied to pixels x[-4] to x[4] to 
obtain pixel values P(x[-4]) to P(x[4]) 5 thereby resulting in a first graph. Additionally, 
function P is applied to pixels y[4] to y[12] to obtain pixel values P(y[4]) to P(y[12]), 
thereby resulting in a second graph. The peak/valley count test is performed to count the 
number of peaks and valleys in the first graph, as well as the number of peaks and valleys 
in the second graph. More than one peak or valley in either one of the two graphs 
indicates that an edge is not likely to be lying along direction D[m=0]. As such, spatial 
interpolation will not be performed along D[m=0]. 
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From the above two specific examples, domain[m,x] (from x[m-4] to x[4]) in line 
110 and domain[m,y] (from y[4] to y[12-2m]) in line 120 vary according to m. That is, 
pixel domains for each interpolation direction are different. 

Moreover, from the above two specific examples, the peak/valley count test can 
be described as follow. Specifically, for D[m] where m is selected from {0, 1, 8}, 
function P is applied to pixels of dom[m,x] (i.e., x[m-4] to x[4]) to obtain pixel values 
P(x[m-4]) to P(x[4]), thereby resulting in a first graph. Additionally, function P is 
applied to pixels of dom[m,y] (i.e., y[4] to y[12-2m]) to obtain pixel values P(y[4]) to 
P(y[12-2m]), thereby resulting in a second graph., graph[m,y]. The peak/valley count 
test is performed to count the number of peaks and valleys in the first graph graph[m,x], 
as well as the number of peaks and valleys in the second graph graph[m,y]. More than 
one peak or valley in either one of the two graphs indicates that an edge is not likely to be 
lying along direction D[m]. As such, spatial interpolation will not be performed along 
D[m]. 

In query step 240, a check is performed to see if D[m] passes the peak/valley 
count test. If affirmative, then step 245 is performed. Otherwise, step 289 is performed. 

In step 245, a pixel value difference test is performed. Specifically, absolute pixel 
value differences (summands) that are used in obtaining A[m] are compared to a pre- 
defined threshold value T. For A[m], these absolute pixel value differences are the 
summands of A[m]: 
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|P(x[m-4])-P(y[m-4])|; 

|P(x[m-3])-P(y[m-3])|; 

|P(x[m-2])-P(y[m-2])|; 

|P(x[m-l])-P(y[m-l])|; 

|P(x[m])-P(y[m])|; 

|P(x[m+l])-P(y[m+l)|; 

|P(x[m+2]) - P(y[m+2])|; 

|P(x[m+3]) - P(y[m+3])|; and 

|P(x[m+4]) - P(y[m+4])|. 
Each of these summands represents the absolute pixel value difference between a pair of 
pixels, one from line 110 and one from line 120. 

If at least one of these quantities exceeds T (pre-defined as a number between o to 
255), then a single absolute difference between the pixel values of two pixels has already 
contributed more than T to A[m]. As such, noise rather than an edge is likely to be 
suggested by pixel values P(x[m-4]) to P(x[m+4]) and P(y[8-m-4]) to P(y[8-m+4]). 
Accordingly, D[m] fails the pixel value difference test if at least one of these quantities 
exceeds T. 

In query step 250, a check is made to verify if D[m] passes the pixel value 
difference test. If affirmative, then step 255 is performed. Otherwise, step 289 is 
performed to replace the disqualified D[m] by a default interpolation direction. 
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In step 255, spatial interpolation at position 1 1 1 is performed along D[m]. 

In step 289, spatial interpolation at position 1 1 1 is performed along a default 
interpolation direction. In the present embodiment, the default interpolation direction 
intercepts effectively the center of position 1 1 1 to form a 45 degree angle with respect to 
resampling line 115. However, as understood herein, in another embodiment, a default 
interpolation direction intercepts effectively the center of position 1 1 1 to form a different 
angle with respect to resampling line 115. 

In step 299, post processing is performed to refine the interpolated image. 
Specifically, the interpolated value at position 1 1 1 is analyzed with respect to the 
neighboring pixel values. Adjustment of the interpolated pixel value at position 1 1 1 is 
performed according to the result of this analysis. 

As understood herein, the cross check test, the peak/valley count test and the pixel 
value difference test need not be performed in the order as outlined in flow chart 200. 
Rather, six different performing orders are possible for performing these three tests. 

Referring now to Figure 3 in view of Figures 1A-C, a system 300 for performing 
image spatial interpolation is shown in accordance with one embodiment of the 
invention. System 300 comprises an evaluation unit 310, a selection unit 320, a cross 
detection unit 330, a peak/valley count unit 340, a pixel value difference count unit 350, 
an interpolation unit 360 and a post-processing unit 370. Specifically, evaluation unit 
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310 and selection unit 320 are parts of a symmetric segment pair detection unit 315. 
Cross detection unit 330 is a part of symmetric segment pairs analysis unit 335. 
Peak/valley count unit 340 and pixel value difference unit 350 are parts of a segment 
analysis unit 345. 

Symmetric segment pair detection unit 315 is coupled to symmetric segment pairs 
analysis unit 335. Symmetric segment pairs analysis unit 335 is coupled to segment 
analysis unit 345. Segment analysis unit 345 is coupled to interpolation unit 360. 
Interpolation unit 360 is coupled to post-processing unit 370. Specifically, evaluation 
unit 310 is coupled to selection unit 320, which is in turn coupled to cross detection unit 
330. Cross detection unit 330 is coupled to peak/valley count unit 340 that is in turn 
coupled to pixel value difference unit 350. Pixel value difference unit 350 is coupled to 
interpolation unit 360. 

Symmetric segment pair detection unit 315 is adapted to assign values to possible 
interpolation directions, wherein each interpolation direction is associated with a pair of 
pixel segments symmetric about position 111. Symmetric segment pair detection unit 
315 is also adapted to detect and select an interpolation direction as the direction whose 
associated segment pair contains the two most similar pixel segments. In the present 
embodiment, symmetric segment pair detection unit 315 comprises evaluation unit 310 
and selection unit 320. 

Evaluation unit 310 is adapted receive pixel values of pixels in lines 110 and 120 
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to generate numbers A[0] to A[8] that are assigned respectively to possible interpolation 
directions D[0] to D[8]. For k = 0 to 8, A[k] quantifies the similarity between two pixel 
segments that are symmetric about position 111. 

Selection unit 320 is adapted to select an interpolation direction D[m] along 
which to perform spatial interpolation for position 111. Specifically, selection unit 320 is 
adapted to select such interpolation direction D[m] whose assigned value A[m] is the 
minimum element of {A[0], A[l], A[8]}. D[m] is then transmitted to cross detection 
unit 330 as a possible candidate for being the interpolation direction to be used by 
interpolation unit 360. 

As understood herein, in the present embodiment, A[m] associated with the 
selected interpolation direction D[m] is the minimum among {A[0], A[l], A[8]}. 
However, depending on the definition of A[m] used, A[m] associated with selected 
interpolation direction D[m] need not be the minimum among {A[0], A[l], A[8]}. For 
example, in another embodiment wherein a different definition of A[k] is used for k = 0 
to 8, A[m] associated with selected interpolation direction D[m] is the maximum among 
{A[0],A[1],...,A[8]}. 

Symmetric segment pairs analysis unit 335 is adapted to analyze all symmetric 
segment pairs associated respectively with D[0] to D[8]. Given the selected symmetric 
segment pair associated with D[m], symmetric segment pairs analysis unit 335 checks to 
see if any of the remaining symmetric segment pairs crosses the selected symmetric 
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segment pair. In the present embodiment cross detection unit 330 is included in 
symmetric segment pairs analysis unit 335 to perform this cross check. However, in 
another embodiment, an extreme value analysis unit 333 can also be incorporated into 
symmetric segment pairs analysis unit 335. In yet another embodiment, extreme value 
333 alone can be incorporated in segment pairs analysis unit 335. 

Cross detection unit 330 is adapted to perform a cross check test on D[m]. By 
performing such cross test, cross detection unit 330 can rule out the problematic scenario 
of interpolating at position 111 along D[m] while two edges cross each other at position 
111. Specifically, the test is performed by cross unit 330 to rule out a scenario wherein 
A[h] of direction D[h] (whose slope has the opposite sign from the sign of the slope of 
D[m]) is slightly difference from A[m]. The cross check test is also performed by cross 
detection unit 330 to rule out a scenario wherein the vertical direction A[4] is slightly 
different with A[m]. D[m] passes the cross check test if no problematic scenario is 
found. As such, if D[m] passes the cross check test, then D[m] is transmitted to 
peak/valley count unit 340 to undergo another test as a viable interpolation direction to be 
used by interpolation unit 360. Otherwise, a default interpolation is passed to peak/valley 
count unit 340 as the interpolation direction to be used by interpolation unit 360. 

Extreme value analysis unit 333 is adapted to detect any one of three cases on the 
graph of A[t] as t varies continuously from 0 to 8. The first case is that A[m] is a 
minimum where m is not equal to 0 or 8, and the slope of A[k] changes sign only once 
from negative to positive. The second case is that A[m] is a minimum where m is 0, and 
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the sign of slope in graph of A[k] remains positive. The third case is that A[m] is a 
minimum where m is 8, and the sign of slope in graph of A[k] remains negative. D[m] is 
disqualified if A[m] does not fit any one of the above three cases. 

Segment analysis unit 345 is adapted to analyze the symmetric segment pair 
associated with D[m]. If any anomaly is found in the analysis, then D[m] is disqualified 
as a viable interpolation direction. In the present embodiment, segment analysis unit 345 
comprises peak/valley count unit 340 and pixel value difference unit 350. Peak/valley 
count unit 340 can be considered as performing first order analysis on the symmetric 
segment pair associated with D[m]. Pixel difference count test unit 350 can be 
considered as performing zero-order analysis on the symmetric segment pair associated 
with D[m]. However, as understood herein, segment analysis unit 345 need not be 
limited as such. For example, in another embodiment of the invention, segment analysis 
unit 345 comprises pixel value difference unit 350, but does not include peak/valley 
count unit 340. In yet another embodiment of the invention, segment analysis unit 345 
comprises peak/valley count unit 340, but does not include pixel value difference unit 
350. In yet another embodiment of the invention, second or higher order of analysis is 
performed on the symmetric segment pair associated with D[m]. 

Peak/valley count unit 340 is adapted to perform an peak/valley count test on 
D[m]. Peak/valley count unit 340 performs the peak/valley count test on two graphs of 
function P obtained by applying P over two domains of pixels, one domain from line 110 
and one domain from line 120. 
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A peak is defined as a point on a graph having a slope on the left side of the point, 
and a slope on the right side of the point. Specifically, the left slope and the right slope 
meet at the peak point. The left slope is positive and larger than a pre-defined threshold; 
the right slope is negative and smaller than a pre-defined threshold. 

A valley is defined as a point on a graph having a slope on the left side of the 
point, and a slope on the right side of the point. Specifically, the left slope and the right 
slope meet at the valley point. The left slope is negative and smaller than a pre-defined 
threshold; the right slope is positive and larger than a pre-defined threshold. 

Specifically, D[m] is assumed to be D[l] as an example. For D[m=l], function P 
is applied to pixels x[-2] to x[4] of dom[l,x] to obtain pixel values P(x[-2]) to P(x[4]), 
thereby resulting in a first graph, graph[l,x]. Additionally, function P is applied to pixels 
y[4] to y[10] of dom[l,y] to obtain pixel values P(y[4]) to P(y[10]), thereby resulting in a 
second graph, graph[l,y]. The peak/valley count test is performed by peak/valley count 
unit 340 to count the number of peaks and valleys in the first graph, as well as the 
number of peaks and valleys in the second graph. 

D[l] passes the peak/valley count test if no more than one peak or valley is 
presented in graph[l,x] and if no more than one peak or valley is presented in graph[l,y]. 
Specifically, more than one peak or valley in either one of the two graphs indicates that 
an edge is not likely to be lying along direction D[m=l]. As such, if more than one peak 
or valley are found in graph[l,x], or if more than one peak or valley are found in 
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graph[l,y], then D[m=l] fails the peak/valley count test. Spatial interpolation will not be 
performed along D[m=l] if D[m=l] does not pass the peak/valley count test. 

To explain with another specific example, D[m] is assumed to be D[0]. 
Specifically, for the example of D[m=0], function P is applied to pixels x[-4] to x[4] to 
obtain pixel values P(x[-4]) to P(x[4]), thereby resulting in a first graph. Additionally, 
function P is applied to pixels y[4] to y[12] to obtain pixel values P(y[4]) to P(y[12]), 
thereby resulting in a second graph. The peak/valley count test is performed by 
peak/valley count unit 340 to count the number of peaks and valleys in the first graph, as 
well as to count the number of peaks and valleys in the second graph. More than one 
peak or valley in either one of the two graphs indicates that an edge is not likely to be 
lying along direction D[m=0]. As such, spatial interpolation will not be performed along 
D[m=0]. 

From the above two specific examples, domain[m,x] (from x[m-4] to x[4]) in line 
110 and domain[m,y] (from y[4] to y[12-2m]) in line 120 vary according to m. That is, 
pixel domains for each interpolation direction are different. 

Moreover, from the above two specific examples, the peak/valley count test can 
be described as follow. Specifically, for D[m] where m is selected from {0, 1, 8}, 
function P is applied to pixels of dom[m,x] (i.e., x[m-4] to x[4]) to obtain pixel values 
P(x[m-4]) to P(x[4]), thereby resulting in a first graph. Additionally, function P is 
applied to pixels of dom[m,y] (i.e., y[4] to y[12-2m]) to obtain pixel values P(y[4]) to 
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P(y[12-2m]), thereby resulting in a second graph., graph[m,y]. The peak/valley count 
test is performed by peak/valley count unit 340 to count the number of peaks and valleys 
in the first graph graph[m,x], as well as the number of peaks and valleys in the second 
graph graph[m,y]. More than one peak or valley in either one of the two graphs indicates 
that an edge is not likely to be lying along direction D[m]. As such, spatial interpolation 
will not be performed along D[m]. 

Accordingly, if D[m] passes the peak/valley count test, then D[m] is transmitted 
to pixel value difference unit 350 to undergo yet another test as a viable interpolation 
direction to be used by interpolation unit 360. Otherwise, the default interpolation is 
passed to peak/valley count unit 340 as the interpolation direction to be used by 
interpolation unit 360. 

Pixel value difference unit 350 is adapted to perform a pixel value difference test. 
Specifically, absolute pixel value differences used in obtaining A[m] are compared to a 
pre-defined threshold value T. For A[m], these absolute pixel value differences are the 
summands of A[m]: 

|P(x[i]) - P(y[j])|; 
|P(x[i]) - P(y[j])l; 
P(x[i]) - P(y[j])|; 
|P(x[i]) - P(y[j])l; 
P(x[i])-P(y[j])l; 
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|P(x[i]) 



P(y[j])l; 



P(x[i]) 



P(yU])l; 



|P(x[i]) 



P(y[j])l; and 



|P(x[i]) 



P(yU])l- 



Each of these summands represents the pixel value difference between two pixels, one 
from line 110 and one from line 120. If at least one of these quantities exceeds T, then a 
single absolute difference between the pixel values of two pixels has already contributed 
more than T to A[m]. As such, noise rather than an edge is likely to be suggested by 
pixel values P(x[m-4]) to P(x[m+4]) and P(y[8-m-4]) to P(y[8-m+4]). Accordingly, 
D[m] fails the pixel value difference test if at least one of these quantities of A[m] 
exceeds T. 

As such, if D[m] passes the pixel value difference test, then D[m] is transmitted 
by pixel value difference unit 350 to interpolation unit 360 as the interpolation direction 
to be used by interpolation unit 360. Otherwise, the default interpolation direction is 
transmitted to interpolation unit 360. 

Interpolation unit 360 is adapted to perform spatial interpolation for position 111. 
This spatial interpolation is performed along D[m] if D[m] has not been rejected by cross 
detection unit 330, peak/valley count unit 340 or pixel value difference unit 350. 
Otherwise, this spatial interpolation is performed along the default direction. 
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Post processing unit 370 is adapted to perform refinement on the interpolated 
image. Post processing unit 370 analyzes the interpolated value at position 1 1 1 with 
respect to the neighboring pixel values. Adjustment of the interpolated pixel value at 
position 1 1 1 is performed according to the result of this analysis. 

As understood herein, coupling order of cross check unit 330, peak/valley count 
test unit 340 and pixel value difference test unit 350 need not be limited to the coupling 
order as described above (cross check unit 330-peak/valley count test unit 340-pixel 
value difference test unit 350). At least five other coupling orders are possible. For 
example, in another embodiment, the coupling order of these units (330, 340 and 350) is 
pixel value difference test unit 350-cross check unit 330-peak/valley count test unit 340. 
That is, pixel value difference test unit 350 is coupled to cross check unit 330 that is in 
turn coupled to peak/valley count test unit 340. 

In all of the embodiments described above (Figures IB, 2-3), for k = 0 to 8, A[k] 
is defined as proportional to the expectation of a random variable, namely function P. 
However, as understood herein, for k = 0 to 8, A[k] need not be limited to the definition 
used above. 

For example, in another embodiment, A[k] is defined as proportional to the 
variance of P. That is, for k = 0 to 8, 

A[k] = Z|P(x[i])-P(y[j]) | 2 , 
i = k-4 to k+4, and j = 8-k-4 to 8-k+4. Using this definition, A[m] associated with the 
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selected direction D[m] is the minimum element of {A[0], A[l], . . ., A[8]}. 

As another example, in yet another embodiment, A[k] is defined as proportional 
to the cross correlation of P. That is, for k = 0 to 8, 

A[k] = Sl/(|P(x[i])*P(y[j]) I), 

i = k-4 to k+4, and j = 8-k-4 to 8-k+4. Using this definition, A[m] associated with the 
selected direction D[m] is the minimum element of {A[0], A[l], . . A[8]}. 

More generally, for the purpose of expressing the degree of the two segments (of 
a symmetric segment pair) being similar, for k = 0 to 8, A[k] need not be limited to the 
form of: 

A[k] = S r = . ctoc f( P(x[i]),P(y[j ]) ), 

i = k-4 to k+4, and j = 8-k-4 to 8-k+4. Rather, for k = 0 to 8, A[k] can be defined more 
generally in the form of: 

A[k]=J(g(x[i]),g(y[j])), 

i = k-4 to k+4, and j = 8-k-4 to 8-k+4, wherein functions J and g can be chosen in 
accordance with the application at hand. 
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For example, in the present embodiment, function g is simply function P acting on 
pixels belonging to the symmetric segment pair associated with D[k], for k = 0 to 8. In a 
different embodiment of the invention, function g is the Fourier transform of P acting on 
pixels belonging to the symmetric segment pair associated with D[k], for k = 0 to 8. In 
another embodiment of the invention, function g is the wavelet transform of P acting on 
pixels belonging to the symmetric segment pair associated with D[k], for k = 0 to 8. In 
yet another embodiment of the invention for motion pictures, function g(x[i]) = q( P(x[i]) ? 
motion_data(x[i])). That is, function g acting at a pixel is equal to a combination q of 
pixel value function P acting at the pixel and the motion data information at the pixel 
belonging to the symmetric segment pair associated with D[k], for k = 0 to 8. As such, 
segment analysis in Figure 2 and segment analysis unit 345 of Figure 3 can be 
implemented using any of the above variations of function g. 

Also as understood herein, in all embodiments of the invention, k need not be 
limited to the range from 0 to 8. For example, in another embodiment, k ranges from 0 to 
2n where n is different from 4. Moreover, the parameters of A[k] need not be as defined 
in the above embodiments. For example, in an alternative embodiment, A[k] is defined 
using i = k-c to k+c, and j = 8-k-c to 8-k+c, where c is different from 4. 

Moreover, as understood herein, for k = 1 to 2n, the number of pixels in 
graph[k,x] and the number of pixels in graph[k,y] need not be limited to that of the 
embodiments described above. Specifically, the numbers of pixels in graph[k,x] and 
graph[k,y] can be chosen to fit the application of the invention contemplated. 
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The foregoing descriptions of specific embodiments of the invention have been 
presented for purposes of illustration and description. They are not intended to be 
exhaustive or to limit the invention to the precise forms disclosed. Obviously, many 
modifications and variations are possible in light of the above teaching. The 
embodiments were chosen and described in order to explain the principles and the 
application of the invention, thereby enabling others skilled in the art to utilize the 
invention in its various embodiments and modifications according to the particular 
purpose contemplated. The scope of the invention is intended to be defined by the claims 
appended hereto and their equivalents. 
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